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METHOD AND SYSTEM FOR INTERACTING WITH A DISPLAY 

FIELD OF THE INVENTION 
Tliis ijivention relates to the field of computer input systems and 
particularly a novel visual method and system for interacting with displays and all 
devices that use such displays. 

BACKGROUND OF THE INVENTION 
5 Remote controllers for TV's, VCR's. cable set top boxes and other 

entertainment appliances have been in common use for quite some time. However, 
these devices have many buttons that often confuse their users. When the devices are 
used to navigate through a menu, tlie hierarchy of menus is often too sequential and 
clumsy. Recently, several manufacturers have introduced "universal remote 
10 controllers" which users have to program for a specific device. Wlien one changes 

batteries or switches televisions re-programming is required. These hassles are often 
annoying to the user. The invention introduces a truly "universal'' remote control that 
one can far more easily replace and/or re-use. 

Also, currently many remote mouse units exist to control computers 
15 and their displays (e.g., projectors or monitors). Some of these still require a flat 

horizontal surface for their tracker. One example is tlie Cordless Wheel Mouse from 
Logitech. Another group of remote mouse controllers are made for use during 
presentations and do not require a surface, but require the user to drag the pointer 
across the screen by operating a trackball or a dial. One example is the RemotePoint 
20 RF Cordless Mouse from Interlink Electronics. The invention provides random 
access to the display space and a far more versatile, facile and intuiti^'e way to 
interact with the display. 

Among prior art, there is a set of patents authored by Lane Hauck et. 
al. of San Diego that define a computer input system for a computer generating 
25 images that appear on a screen. These are listed in the References Cited and 
discussed in some detail below. 

Patent 5,181,015 is the initial patent describing a method and 
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apparatus for calibrating an optical computer input system. The claims focus 
primarily on the cahbration for facilitating the ahgnment of the screen image. Patent 
5,489,923 carries the same title as the first patent (#5,181,015) and 
is similar in its content. Patent 5,515,079 appears to have been written when the 

5 inventors wanted to claim the computer input system, rather than their prior and 

subsequent more specific optical input and calibration systems. We consider this and 
what appears to be its continuation in Patent 5,933,132 to be the most relevant prior 
ait to this invention. This patent defines a computer input system and method based 
on an external light source pointed at the screen of a projector. In the continuation 

1 0 patent 5,933,132, a method and apparatus for calibrating geometrically an optical 

computer input system is described. This is to take care of the geometric errors that 
appear in relating the image of tlie projection to that of the display. However, this 
correction relies exclusively on the four comers of a projected rectangle and thus 
compensates only partially for the most obvious errors, and thus still provides a 

15 limited correction. Patent 5,594,468 describes in a detailed and comprehensive 

manner additional means of calibrating - by which the authors mean determining the 
sensed signal levels that allow the system to distinguish between the user generated 
image (such as the light spot produced by a laser pointer) and the video source 
generated image that overlap on the same display screen. Patent 5,682 J 81 is another 

20 improvement on 5,515,468 and is mainly concerned with superimposing an image 
based on the actions of the external light source on the image produced by the 
computer. This is done to allow the user holding the light source to accentuate the 
computer image. 

All of the cited patents describe methods based on external proprietary 
25 hardware for image registration and signal processing. Because of the nature of the 
image acquisition, the methods used by the said invention of prior art differ 
significantly from those of this invention, which uses off-the-shelf standard hardware 
and software routines as embodiments of the methods described and claimed to 
combine them into a seamless human-machine interaction system. Moreover, the 
30 input system in prior art fiinctions only with a specific set of pointing devices. No 
method for other pointing devices is provided. No correction based on the actual 
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mouse position registered by the camera is provided. No method for a pointing 
device that is used outside of the real display space is provided. 

91 TMMAT^ Y OF THE INVENTION 
The hardware elements of a simple implementation of the invention 
consist of a projector, camera, and a pointing device such as a laser pointer. Some of 
the many intended applications of this invention is as a replacement for a computer 
mouse pointer and as a replacement for a computer pen or stylus. The invention can 
5 replace a common PC mouse, or a menu-driven remote control device with an 

arbitrary pointing device, such as a laser pointer or another light source or another 
pointing device with recognizable characteristics, e.g., a pen. a finger worn cover, 
e.g., thimble, a glove or simply the index finger of a hand. By implementing a system 
defined by this invention, one can use a pointing device (e.g., a laser pointer) during a 
1 0 computer presentation not only to point to specific locations on the screen proj ected 
by an LCD projector or a rear projection screen display, but also to interact with the 
computer to perform all fiinctions that one can ordinarily perform with a PC mouse or 
remote control for the display. The invention can also be interfaced with and operate 
in tandem wdth voice-activated systems. The data fi-om the camera can be processed 
15 by the system to (1) determine the position of the location of the pointing device (e.g., 
the reflection of the laser pointer or the position of the thimble) on the display, (2) 
position the mouse pointer at the corresponding screen position, and (3) "click" the 
mouse when a programmable pre-determined pointer stroke or symbol is detected, 
such as a blinking laser spot or a tap of the thimble. All of these features allow the 
20 user unprecedented convenience and access to a vast variety of programmable remote 
control fianctions with only an ordinary pointing device. In the same scenario, the user 
can also annotate the presentation or create a presentation on any ordinary board or 
wall surface, by using the pointing device as a stylus. A remote control application of 
the invention in a home entertainment setting uses a laser pointer. 
25 Displays, light sensors or cameras and pointing devices of the 

invention can be selected from a variety of commercially available hardware devices. 
No special hardware is required. The invention also defines methods of using the 
said hardware to create a seamless visual interaction system. The methods, too, can 
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work with a variety of display, camera, and pointing devices. Future display devices 
could incorporate a camera within the display to achieve this type of fonctionality in a 
single device. 

The invention can thus be used as a general-puipose tool for visual 
5 interaction with a PC (or PC-like device or a TV projection screen) through its 

display using only a common pointing device, such pointing device not having to 
contain any special mechanical, electronic or optical mechanism or computing or 
communication apparatus. The invention can also work in tandem with a common 
PC mouse, overriding the common mouse only when the user points the designated 
1 0 pointing device onto the projected display area. 

BRIEF DESCRIPTION OF THE DRAWINGS 
The description herein makes reference to the accompanying drawings 
wherein like reference numerals refer to like parts throughout the several views, and 
wherein: 

15 Figure 1 shows hardware elements used in the present invention 

including a projector, camera and a pointing device, such as a laser pointer; 

Figure 2 shows a user with the pointing device to annotate a 
presentation on a wall surface; 

Figure 3 shows a user with a remote control to control entertainment 

20 components on a wall surface; 

Figure 3a shows an enlarged view of the entertainment components; 
Figure 4 shows elements of the system using a tliimble as the pointing 

device; 

Figures 5-11 show the visual steps of the system; 
25 Figure 12 shows one possible arrangement of the elements of the 

system using a rear projection display; 

Figure 13a-13b are two examples of arrangements of the system where 
a light sensor cannot view an actual display; 

Figure 14 is a flowchart outlining the method for detecting a real 

30 display; 

Figure 15 is a flowchart outlining the method for registering the 
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pointing device in a real display case; 

Figure 1 6 is a flowchart outlining the method for detecting a virtual 

display; 

Figure 17 is a flowchart outlining the method for registering the 
5 pointing device in a virtual display case; 

Figure 1 8 is a flowchart outlining tlie method for computing the 
mapping between a display space registered by the light sensor and the computer 

display; 

Figures 19a-l 9d show a series of frames of the reflection of the 
10 pointing device in a lit room; 

Figures 19e-19h show a series of frames of the reflection of the 
pointing device in a dark room; 

Figure 20a shows a computer display image; 

Figure 20b shows an image of display from a light sensor; 
15 Figure 20c shows an image-display mapping; 

Figure 21a shows a display space image in a distorted case; 

Figure 21b shows an image of display from a light sensor in a 

distorted case; 

Figure 21c shows an image-display mapping in a distorted case; 
20 Figure 22a-22c shows the correspondence between the image of a 

virtual display and the computer display; 

Figures 23a-23c shows the correspondence between the position of the 
pointing device in relation to the image of the real display and the computer display; 

Figure 24a shows an acceptable positioning of the computer pointer; 
25 Figure 24b shows an unacceptable positioning of the computer 

pointer; 

Figures 25a-25d illustrates steps for selecting an item on the display; 
Figure 26 is a flowchart outlining the method for selecting an item; 
Figure 27 is a perspective view of a light pen; and 
30 Figure 28 is a flowchart summarizing the system operation, which is 

the background or backbone process of the system. 
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DESCRIPTION OF THE PREFERRED EMBODIMENT 
This invention relates to the field of computer input systems. The 
hardware elements of a simple implementation of the invention are shown in Figure 
1. Hardware elements of tlie invention consist of a projector 12, camera 14, and a 
5 pointing device such as a laser pointer 16. Some of the many intended applications 
of this invention is as a replacement for a computer mouse pointer and as a 
replacement for a computer pen or stylus. The invention can replace a common PC 
mouse, or a menu-driven remote control device with an arbitrai-y pointing device, 
such as a laser pointer 1 6 or another light source or another pointing device with 
10 recognizable characteristics, e.g., a pen, a finger worn cover, e.g., thimble, a glove or 
simply the index finger of a hand. By implementing a system defined by this 
invention, one can use a pointing device (e.g., a laser pointer) during a computer 10 
presentation not only to point to specific locations on the screen 32 projected by an 
LCD projector or a rear projection screen display, but also to interact with the 
1 5 computer 1 0 to perform all functions that one can ordinary perfomi with a PC mouse 
or remote control for the display. The invention can also be interfaced with and 
operate in tandem with voice-activated systems. The data fi-om the camera 14 can be 
processed by the system to (1) determine the position of the location of the pointing 
device (e.g., the reflection of the laser pointer 16 or the position of the thimble) on the 
20 display 32, (2) position the mouse pointer at the corresponding screen position, and 
(3) "click" the mouse when a programmable pre-determined pointer stroke or symbol 
is detected, such as a blinking laser spot or a tap of the thimble. All of these features 
allow the user 1 8 unprecedented convenience and access to a vast variety of 
programmable remote control fimctions with only an ordinary pointing device. In the 
25 same scenario, the user 1 8 can also annotate the presentation or create a presentation 
on any ordinary board or wall surface, by using the pointing device as a stylus. 
Figure 2 illustrates one of the many scenarios where an LED light pen 20 can be used 
to control a computer during a presentation. The LED light pen 20 can also annotate 
the presentation. A remote control application of the invention having in a home 
30 entertainment setting using a laser pointer 1 6 is illustrated in Figure 3 . Figure 3a 
shows examples on a display wall 32 or projector 12 including a PC desktop 22, 
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audio 24, the Internet 26 and TV or cable 28. A mouse pointer at a laser light spot 30 
is also shown. 

The display, light sensor or camera that can register the display image 
and the pointing device or its reflection on the display, and a pointing device that can 
5 be registered by or produces recognizable characteristics that can be registered by the 
light sensor or camera can be selected from a variety of commercially available 
hardware devices. No special hardware is required. The invention also define 
methods of using said hardware to create a seamless visual interaction system. The 
methods, too, can work with a variety of display, camera, and pointing devices. 
1 0 Future display devices could incorporate a camera within the display or on the 

associated projection apparatus to achieve this type of functionality in a single device. 

The invention can thus be used as a general -purpose tool for visual 
interaction with a PC (or PC-like device or a TV projection screen) through its 
display using only a common pointing device, such pointing device not having to 
1 5 contain any special mechanical, electronic or optical mechanism or computing or 
communication apparatus. The invention can also work in tandem with a common 
PC mouse, overriding the common mouse only when the user points the designated 
pointing device onto the projected display area. 

Starting with Figure 4, which shows the physical elements of the 
20 invention including the computer 10 connected to the display, a light sensor 14, the 
display 12, a colored thimble 30 as the pointing device. Figures 4-1 1 illustrate the 
concepts behind the invention in relation to a specific example application using a 
. simple colored thimble pointer step by step. Elements of the system are specific to 
the example application. In Figure 4, a colored tliimble is the pointing device. In 
25 Figure 5, a projector projects the display of the PC onto a wall. In Figure 6, a camera 
views the projected PC display. In Figure 7, the system algorithms establish the 
correspondence between the device display (left) and the projected image as it is 
"seen" by the camera (right). In Figure 8, the system instructs the user to register 
his/her pointing device against a variety of backgrounds. During this registration 
30 process, the system compiles a list of characteristics of the pointing device, e.g. its 
color, shape, motion patterns, etc., which can be used later to locate the pointing 
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device. In Figure 9, the system algorithms take control of the PC mouse only when 
the camera registers sees the registered pointing device in the display ai-ea. In Figure 
10, the system steers the mouse pointer to the display location pointed to by the laser 
pointer. In Figure 1 1 , the system sends a command that ''clicks" the mouse when the 
5 pointing thimble is held steady for a programmable length of time, or based on some 
other visual cue, e.g., tap of tlie thimble registered visually, or external cues by v^ay 
of interaction with an external system, e.g., by sound or voice command of the user. 
In the example application given in Figures 4-1 L the system serves the purpose of a 
one-button general puipose remote control when used with a menu displayed by or in 

1 0 association with the device being controlled. The menu defined on the visible display 
sets the variety of remotely controlled functions, without loading the remote control 
itself with more buttons for each added functionality. Moreover, the system allows 
the user random access to tlie display space by simply pointing to it, i.e., there is no 
need to mechanically "drag" the mouse pointer. 

1 5 Pushing menu buttons on the display screen with a simple thimble 

pointer 30 is certainly only one of the applications of this invention. One can also 
imagine a PC, a TV, a telephone, or a videoconferencing device controlled remotely 
by a pointing device, e.g., laser pointer, that is pointed onto a projected image 
corresponding to the graphical user interface (GUI) of the device. In this scenario, 

20 the monitor or CRT or display apparatus is replaced by a projector 12, and the display 
is thus a projection 32 is on a surface (such as a wall). The viewable image size can 
be quite large without the cost or the space requirements associated with large display 
devices. Moreover, the pointing device (e.g., laser pointer 16) allows the user 
mobility and offers many more functions than an ordinary remote control can. Also, 

25 the laser pointer 1 6 is the size of a pen and is smaller and simpler to use than a 

remote control. As many people who have ever misplaced the remote control code 
for their TV's or VCR's can appreciate, this new device can be single button universal 
remote control with no preprogramming requirement. In fact. Figure 3 illustrates this 
scenario. 

30 Many types of displays that are currently available or those that will be 

available can be used. This includes, but is not limited to LCD projectors and rear 
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projection displays, as well as CRT's. In case of the LCD projector, it makes 
practical sense to position the camera 14 near the projector 12. In case of a rear 
projection display 32, one option is to have the camera 14 view the backside of the 
visible display. Figure 12 illustrates a possible arrangement of the system elements 
5 when used by a rear projection display. The pointing device or its reflection must be 
visible to the light sensor. A miiTor is indicated at 34, the viewable display is 
indicated at 32, and reflection of the pointing device on the display is indicated at 36. 

The light sensor 14 should be capable of sensing all or part of the 
display and the pointing device 1 6 or its effect 36 on the display. A one-dimensional 
10 light sensor could be used with a very simple and constrained system, but generally a 
two dimensional (area) light sensor could be used with a two dimensional display, 
although other arrangements are also possible. The elements of a light sensor are 
generally capable of registering a particular range of light frequencies. The light 
sensor may be composed of multiple sensors that are sensitive to several different 
15 ranges of light frequencies, and thus be capable of sensing multiple ranges (or colors) 
although that is not a requirement. The sensor needs to deliver data, which can be 
used by the method described below to detect the pointing device 16 or its reflection 
on or outside the display 32. In most cases, the sensor needs to be capable of sensing 
the pointing device 16 in all areas of the display 32. In this sense, it is preferable, 
20 under most circumstances for the light sensor 14 to be capable of viewing all of tlie 
display. However, this is not a limitation, as subsequent sections of this document 
make clear. Best resolution would.be achieved with a sensor whose field of view 
exactly matches the whole display. In some cases, it may be preferable to use a 
pointing device 16 that emits or reflects light or other electromagnetic waves 
25 invisible to the human eye. In this case, if the mentioned invisible waves is a 

characteristic that the system relies on to distinguish the pointing device from other 
objects in its view, the light sensor must be able to register this characteristic of the 
pointing device or its reflection. 

One of the distinguishing characteristics of the invention is in its 
30 versatility of allowing for a wide range of pointing devices 16. The system allows the 
user to select any convenient appropriate pointing object that can be registered by the 
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light sensor. The more distinguishable the object, the better and faster the system 
performance will be. A light source is relatively easy to distinguish with a simple set 
of computations, so a light source may initially be the preferred embodiment of the 
invention. A laser pointer 16 or other light source is a potential pointing device that 
5 can be used. The system can also use many other types of visible (e.g., pen with LED 
light) or invisible (e.g., infrared) light sources so long as they are practical and can be 
registered by the light sensor as defined siqjra. 

However, the invention is by no means hmited to using light sources 
as pointing devices. A thimble 30 with a distinguishing sliape or color that can be 
10 picked up by the light sensor is another potential pointing device. As the 

performance of tlie computer on which the computations are performed increases, the 
invention will accommodate more and more types of pointing devices, since virtually 
every object will be distinguishable if sufficiently sophisticated and lengthy 
computations can be performed. 
1 5 Therefore, there are no limits on the types of pointing devices the 

system of this invention can use. Note that the name "pointing device" is used very 
loosely. It has already been mentioned that a pointing device 16 can be the index 
finger of one's hand. There are otlier ways of pointing that are more subtle and do 
not involve translational re-positioning of the pointing device. Imagine for example a 
20 compass that rotates and points to different directions. The length or color of the 
needle can be defining a point on the display. Also imagine a pointing mechanism 
based on the attitude of an object (such as the presentation of a wand, one's face or 
direction of gaze). The system of this invention can be used with such pointers, so 
long as the light sensor is capable of registering images of the pointing device, which 
25 can be processed to determine the attitude or directions assumed by the pointing 
device. 

So far, only absolute positioning has been implied. This is not a 
limitation of the invention, either. Although in the examples shown in Figures 4- 1 L 
it makes sense to use the pointer as an absolute addressing mechanism for the display, 
30 it may also be convenient to use a pointer as a relative addressing mechanism. In 
fact, many current computer mouse devices utilize relative positioning. 
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There are two cases for detecting the display and the pointing device 
both of which can be accommodated by this invention. The first is when the light 
sensor can view the same display space that is being viewed by the user 18. This 
would be the projected image screen 32 or the monitor, which we call "the real 
display." The second case is somewhat more interesting. This is the case where the 
light sensor cannot view the actual display, possibly because it is not in the field of 
view of the light sensor. Consider for example, that the light sensor is mounted on 
the display itself. Two examples are depicted in Figures 13a and 13b. Figure 13a 
shows an example with a handheld computer 40 having a display 32 and a light 
sensor or camera 14. A colored thimble 30 is used as a pointing device. Figure 13b 
shows an example with a TV console 42. The user 1 8 is using a colored stick or pen 
as the pointing device. The range of allowed positions for the pointing device (all of 
which should be in the field of view of the sensor) defines "the virtual display space." 
The invention can still be employed, even though the display itself is not visible to 
the light sensor 14. We call the range of allowed positions for the pointing device (all 
of which should be in the field of view of the sensor) "the virtual display space." In 
botli of these cases, it is still necessary that the pointing device or its reflection on the 
display is in the field of view of the light sensor 14, at least when the user is using the 
system. 

The method for tlie real display case is outlined in the flowcharts in 
Figures 14 and 15. In Figures 14 and 15, after the start step 50, two alternate paths 
are presented, each leading to step 60. The system can follow either path, namely 52, 
or 54. Step 54 then proceeds to steps 56 and 58. Step 52 details the user of the 
system first turning the display on, followed by Uie system finding the display area 
using the image firom the light sensor, based on the characteristics of the display or 
the known image on the display. On the other hand, the user or the system can turn 
the system off, and with the light sensor capture a fi-ame of the display (step 54), then 
turn the display on and capture a frame of the display space (step 56). The system 
then locates the display by examining the difference between the two fi-ames (step 
58). After these steps the user or the system can adjust the light sensor position and 
sensing parameters for best viewing conditions (step 60) and then check whether the 
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results are satisfactory. If not satisfactor>', the user or the system returns to step 50. 
If the results are satisfactory, the system defines the borders of the display in the 
image captured by the light sensor as continuous lines or curves (step 64), and 
outputs or stores the borders of tlie display as they are captured by the light sensor, 
5 their visual characteristics, location and curvature (step 66). Step 68 continues to 

pointing device registration. Alternately, the system may proceed to step 132, if the 
pointing device has already been characterized or registered. The display image used 
during these procedures may be an arbitraiy image on the display or one or more of a 
set of calibration images. Step 70 instructs user to register the pointing device he/she 
1 0 will use. The user may select a pointing device from a list or have the system register 
the pointing device by allowing the pointing device to be viewed by the light sensor. 
Between step 70 and step 80, two alternate paths are presented. Either path can be 
followed. In step 72 the user is instructed to point the pointing device to various 
points on the display. The system then captures one or more frames of the display 
1 5 space with the pointing device. Alternately, steps 74, 76, and 78 can be followed. 

The system then can capture a frame of the display space without the pointing device 
(step 74), capture a frame of the display space with the pointing device (step 76), and 
locate pointing device by examining the difference between the two (step 78). After 
these steps the user or the system can adjust the light sensor or camera position and 
20 viewing angle as well as the sensing parameters for the best viewing conditions (step 
80) and then check whether the results are satisfactory (step 82). If not satisfactory, 
the user or the system returns to step 70. If the results are satisfactory, the system has 
been able to determine the distinguishing characteristics of the pointing device which 
render it distinct from the rest of the display by analyzing the images recorded by the 
25 light sensor or camera against an arbitrary image on the display or against a set of 
calibration images and adjusting the light sensor or camera position, viewing angle 
and sensing parameters for optimum operation (step 84). In step 88, distinguishing 
characteristics of the pointing device against a variety of display backgrounds are 
outputted or stored. In step 86 the system continues to computing the mapping 
30 between the display space registered by the light sensor and the coinputer display 
132. 



01/52230 — PCT/USOl/00776 



13 

The method for the virtual display case is defined by the flowcharts in 
Figures 1 6 and 1 7. In Figures 1 6 and 1 7, after the start step 90, two alternate paths or 
processes are presented each leading to step 1 00. The system can follow either path, 
namely 92 or 94. Step 94 then proceeds to step 96 and 98. The system or the user 
can turn the display on, at which point the system instructs the user to point the 
pointing device to a convenient or specific area or location of the display (e.g., 
center). Using the image from the light sensor or camera, the system locates the 
pointing device based on the known characteristics of the pointing device (step 92). 
On the other hand, the user can be instructed to first hide the pointing device, and 
using the light sensor or camera, the system captures a frame of the display space 
(step 94), second the user can be instructed to point the pointing device to a 
convenient or specific location of the display, and using the light sensor or camera, 
the system captures a frame of the display space (step 96), third the system locates the 
pointing device by examining the difference between the two frames (step 98). After 
these steps the system or the user can adjust the light sensor position, viewing angle 
and sensing parameters for best viewing conditions (step 100) and then check 
whether the results ai e satisfactory' (step 1 02). If not satisfactory, the user or the 
system returns to step 90. If the results are satisfactory, in step 1 04 the system 
instructs the user to point with the pointing device to the borders and/or various 
locations of the display and captures frames with the light sensor or camera. Then, 
the system defines the borders of the display space in the image captured by the light 
sensor or camera as continuous lines or curves (step 106). The borders of the display, 
as they are captured by the light sensor or camera, their visual characteristics, 
location, and curvature (step 108) are outputted or store(d. Step 1 10 continues to 
pointing device registration. Note that the steps 1 12 through 1 18 can be skipped if 
distinguishing characteristics of tlie pointing device have already been computed to a 
satisfactory degree or are known a priori. Moreover, the order of the processes (92 
through 1 10) and (1 12 through 120) may be changed if it is desirable to register the 
pointing device first and then set the display space. Step 1 12 instructs user to point 
with the pointing device to the borders and/or various locations of the display. The 
system captures frames with the light sensor or camera. After the steps, the user or 
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the system can adjust the light sensor position, viewing angle, and sensing parameters 
for best viewing conditions (step 1 14). The user or the.system then checks whether 
the resuhs are satisfactoi^ (step 1 16). It not satisfactor)', the user or the system 
returns to step 114. If the results are satisfactory, the system determines the 
5 characteristics of the pointing device that distinguish it from the rest of the virtual 
display by observing it via the light sensor or camera against the background of the 
virtual display. The system or user can then adjust light sensor position, viewing 
angle, and sensing parameters for optimum operation (step 118). Distinguishing 
characteristics of the pointing device against the variety of display backgrounds are 
1 0 outputted or stored (step 1 20). Having completed the steps 1 1 8 and 1 20, the system 
can continue to compute the mapping between the display space registered by the 
light sensor and the computer display (step 122). 

In both the real and the virtual display space cases, the system uses a 
particular method for detecting the display or the virtual display space. In tiie first 
1 5 case, usually, the actual image that is on the display is known to the system, so the 
light sensor can be directed to locate it, automatically by way of (i) cropping a large 
high resolution image, or (ii) a pan/tilt/zoom mechanism under the control of the 
system. Alternately, the user can adjust the viewing field of the sensor. The system 
will operate most optimally if the field of view of the light sensor contains the whole 
20 display, as large as possible, but without any part of the display being outside of the 
field of view. In the second case as illustrated in the flowcharts of Figures 16 and 17, 
the light sensor or camera cannot register the real display, but the virtual display 
space. In order to operate successfully, the light sensor must have the pointing device 
in its field of view at all or nearly all times that the user is employing the system. In 
25 this case, too, the system needs to compute the dimensions of the space where the 
pointing device will be, i.e., the virtual display space. The system could be set 
automatically based on the recognition of objects in the virtual display space, and 
their relative dimensions, especially in relation to the size of the pomting device. 
Alternately, the user can manually do the same by adjusting the position and the field 
30 of view of the light sensor or camera. The virtual display case may call for a relative 
address scheme, rather than an absolute addressing scheme. Relative addressing may 
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be practical in this case since the user is not necessarily pointing to the actual space 
where he/she desires to point to or cause the computer's pointer to be moved to. 

Following the establishment of the correct field of view for the real 
display or the virtual display space, at least one view of the same is registered. This 
5 is often in the form of a snapshot or acquired data or image frame from the light 

sensor. The related data output from the light sensor can be formatted in a variety of 
ways, but the method should be able to construct a one or two-dimensional image 
from the acquired data which maintains the spatial relationship of the picture 
elements of the light sensor (and consequently the scene). This one snapshot may be 
1 0 followed by one or more additional snapshots of the real or the virtual display space. 
One example may involve capturing two images, one with the display on and the 
other with the display off This may be an easy way of finding the location and 
boundary contours of the display, as well. Additional snapshots could be taken but 
this time with or without the pointing device activated and in the view of the light 
1 5 sensor. The user may be instructed to point to different locations on the display to 
register the pointing device, its distinguishing characteristics, such as the light 
intensity it generates or registers, at the light sensor, its color, shape, size, motion 
characteristics etc. (as well as its location) against a variety of backgrounds. Note 
that the acquisition of the image with and without the pointing device may be 
20 collapsed into a single acquisition, especially if the characteristics of the pointing 
device are already known or can readily be identified. Note that the capture of 
images can happen very quickly without any hmnan intervention in the blink of an 
eye. The most appropriate time to cany out these operations is when the system is 
first turned on, or the relative positions of its elements have changed. This step can 
25 also be carried out periodically (especially of the user has been idle for some time) to 
continuously keep the system operating in an optimum manner. 

Using the images captured, the system determines the outline of the 
display or the virtual display space, and the characteristics of the pointing device that 
render it distinguishable from the display or the virtual display space in a way 
30 identifiable by the system. The identification can be based on one or more salient 

features of the pointing device or its reflection on the display, such as but not limited 
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to color, (or other wavelength-related information), intensity (or luminance), shape or 
movement characteristics of the pointing device or its reflection. If the identified 
pointing device (or reflection thereof) dimensions are too large or the wrong size or 
shape for the computer pointer, a variety of procedures can be used to 
5 shrink/expand/or reshape it- Among the potential ways is to find a specific boundary 
of the pointing device (or its reflection) on the display. Another method of choice is 
to compute the upper leftmost boundary of the pointing device (for right handed 
users), or the upper rightmost boundary of the pointing device (for left handed users), 
or the'center of gravity of the pointing device or its reflection. A procedure based on 
10 edge detection or image moments, well-known to those skilled in the art of image 

processing, can be used for this, as well as many custom procedures that accomplish 
the same Ir corresponding results. Figures 19a-19h illustrate how the reflection of a 
pointing device (in this case laser pointer light source pointed towards a wall) can be 
identified traced by use of center of gravity computations. The figures show this 
15 under two conditions, namely in a lit (Figures 19a-19d) and a dark room (Figures 
19e-19h). The position of the center of the light spot (marked with an "x" ) can be 
computed at each frame or at selected number of firames of images provided by the 
light sensor. The frames in the figure were consecutively acquired at a rate of 30 
frames per second and are shown consecutively from left to right in the figure. 

A flowchart of the method for establishing the correspondence 
between the position of the pointing device in relation to the display as it is registered 
by the light sensor and its position in relation to the computer display space in case of 
a real or virtual display is shown m Figure 18. First, step 132 divides the display 
space into the same number of regions as those of the computer display using the 
25 information from tlie borders of the display space. (The output in step 66 or 108 is 

used as input 130 to step 132.) In step 134 the system establishes the correspondence 
between the real or virtual display space observed by the light sensor and the 
computer display region by region, and makes necessai7 adjustments to the 
boundaries of individual regions as necessar>'. Then m step 138, the system can make 
30 adjustments to the mapping computed in step 1 34 by using the information from the 
position of the pointing device previously registered by the user and the images 
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captured when the user pointed the pointing device to the regions of the display as 
instructed by the system. This can further improve the mapping between the image 
space registered by the light sensor and tlie computer display. (The outputted data 
from steps 88 or 120 is input 136 to step 138.) Images captured with the pointing 
device pointing to various regions of the display (step 140) is also input to step 138. 
Note that step 138 may be skipped, however, if the mapping computed in 134 is 
sufficient. Mapping between the display space and the computer display is outputted 
(step 144). The user continues to system operation in step 142. System operation is 

illustrated in Figure 28. 

A distinction must be made for purposes of clarity between the display 
or the virtual display space that is registered by the light sensor and the computer 
display space: The computer display space is defined by the computer or the device 
that is connected to the display. It is defined, for example, by the video output of a 
PC or a settop box. It is in a sense the "perfect image" constructed fi-om the video 
output signal of the computer or the visual entertainment device connected to the 
display. The computer display space has no distortions in its nominal operation and 
fits the display apparatus nearly perfectly. It has the dimensions and resolution set by 
the computer given tlie characteristics of the display. As an example, if you hit the 
"Print Screen" or "PrtScr" button on your PC keyboard, you would capture the image 
of this computer display space. Tliis is also what is depicted in Figure 20a as a 9 x 12 

computer display. 

The display or the virtual display space that is registered by the light 

sensor, on the other hand, is a picture of the display space. This is also what is 

depicted in Figure 20b. Being a picture registered by an external system, it is subject 
25 to distortions introduced by the camera or the geometry of the system elements 

relative to each other. A rather severely distorted rendition of the display space 

obtained from the light sensor is depicted in Figure 21b. 

Interaction with the display and/or the device that the said display is 

connected to requires that a correspondence be established between the display space 
30 (whether real or virtual) as it is registered by the light sensor and the computer 

display space. 



15 
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This correspondence between the actual display space and the 
registered real display space can be established (i) at system start time, or (ii) 
periodically during system operation, or (iii) continuously for each registered image 
frame. In Figures 20a-20c, a simple example of how the said correspondence can be 
5 established is illustrated. For the purpose of this example, assume that the actual 

display space is composed of a 9 x 12 array of picture elements (pixels) and that the 
light sensor space is 18 x 24 pixels. In this simple case, the display falls completely 
whhin the light sensor's view in a 1 6 x 21 pixel area, and is a perfect rectangle not 
subject to any distortions. This 16x21 pixel area can be partitioned into a 9 x 12 
10 grid of the display space, thus establishing con-espondence between the actual (9 x 12 
pixel display) and the image of the display acquired by the light sensor. 

In practical operation, the image(s) of both the display and the 
pointing device (or its reflection) will be subject to many types of distortions. Some 
of these distortions can be attributed to the geometry of tlie physical elemems, such as 
1 5 the pointing device, the display, the viewing light sensor, and the projector (if 

applicable). Further distortions can be caused by the properties of the display surface 
and imperfections of the oplical elements, e.g., lens, involved. In cases where these 
distortions are significant, for successful operation of the system, their effects need to 
be considered during the establishment of the display-image correspondence. An 
00 illustrating example is given m Figures 21a-21c. Although a more complex 

correspondence relationship exists in this severely distorted case, the outline of the 
procedure for determining it remains the same. At least one picture of the real 
display space is taken. The method searches the real display space for a distorted 
image of the computer display space (which is known). The nature of the distortion 
25 and the location of the fit can be changed during the method until an optimum fit is 
found Many techniques known in the art of image and signal processing for 
establishing correspondence between a known image and its distorted rendition can 
be used. Furthermore, the use of one or more special screen images can make the 
matching process more effective in the spatial or the frequency domain (e.g., color 
30 block patterns or various calibration images, such as, but not Hmited to the EIA 

Resolution Chart 1956, portions of the Kodak imaging chart, or sinusoidal targets). 
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Another simplifying approach is to take two consecutive images, one with the display 
off and the other with the display on. The difference would indicate the display space 
quite vividly. The various light sources (overhead lights, tabletop lights, sunlight 
through a window) can introduce glares or shadows. These factors, too, have to be 

taken into consideration. 

The captured image(s) can be processed further to gauge and calibrate 
the various settings of the display and the light sensor. This information can be used 
to adjust the display and the light sensor's parameters for both the optimum viewing 
pleasure for the user and the optimum operation of the system. 

If the system is being used without tlie light sensor having in its field 
of a view the display (i.e., the virtual display space case), the image captured by the 
light sensor is the rendition of the environment from which the pointing device will 
be used. In this case establishing correspondence between the virtual display space 
and the computer display requires a different approach illustrated in Figures 22a-22c. 
1 5 In the illustration (Figures 22a-22c), the computer display is a 9 x 1 2 

pixel area as before. The light sensor cannot view the real display (for reasons such 
as those depicted in Figure 13), but instead views the so-called virtual display - the 
vicinity of where the designated pointing device can be found. The reach of the 
pointing device in the user's hands defines the virtual display area. This range can be 
20 defined automatically or manually during the setup of the system. The user can point 
to a set of points on the boundary of the virtual display area xyhile being guided 
through a setup routine 92, 94, 96, and 104. The user can also be guided to point to 
other regions of the computer display, such as the center for better definition of the 

virtual display space 104, 1 12. 

25 To successfully use the pointing device, in addition to the 

correspondence between the computer display space and the real or virtual display 
space registered by the image sensor, one also needs to establish the correspondence 
between the pointing device and computer display locations. For this, the method for 
detecting the display and the pointing device on or in relation to the display, must be 

30 combined with the method for establishing correspondence between the computer 
and registered display spaces described in this section. An illustrative example is 
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given in Figures 23a-23c wherein establishing correspondence between the position 
of the pointing device (or its reflection on the display in this case) in relation to the 
image of the real display and the computer display and positioning of the pointer 
accordingly in a complex (severely distorted) case. In this case, the pointing device is 
5 a laser pointer. The detected position of the reflection of the light from laser pointer 
is found to be bordering the display locations (3,7) and (4,7) in Figure 23b. The 
center of gravity is found to be in (4,7) and thus the pointer is placed inside the 
computer display pixel location (4,7) as illustrated in Figure 23c. 

The method for correcting the offsets between (i) the position of the 
10 pointing device or reflection thereof on the display as observ^ed by the user or by the 
light sensor, and (ii) the position of the pointer on the computer display space applies 
only to the real display case. This correction need not be made for the virtual display 
case. Ideally, if all the computations carried out to establish correspondence between 
the image of the real display registered by the light sensor and the computer display 
1 5 were completely accurate, the position of the pointer device (or reflection thereof) 

and the position of the pointer on the screen would coincide. This may not always be 
the case, especially in case of more dynamic settings, where the light sensor's field of 
view and/or the display location change. In Figures 24a-24b, an acceptably accurate 
(Figure 24a) and an unacceptably inaccurate (Figure 24b) positioning of the 

20 pointer are shown. 

Generally speaking, there are three sources of errors. These are: 

(1) Static errors that arise due to 

a. inaccuracy in the correspondence mapping, and 

b. inaccuracy due to quantization errors attributable to 
9 5 incompatible resolution between the display and the 

light sensor. 

(2) Dynamic errors that arise out of the change in the geometry of 
the hardware, as well as the movement of the pointing device 
or its reflection on the display. 

30 (3) Slow execution of the system where the computations do not 

complete in time and the computer pointer lags behind the 
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pointing device. 

These errors can be corrected by capturing an image of the display 
with the pointer on the display, identifying (i) the location of the pointing device on 
the display and (ii) the location of the computer's pointer representation (e.g., pointer 
arrow) in the captured image, identifying the disparity between (i) and (ii) and 
correcting it. A variety of known methods, such as feedback control of the 
proportional (P), and/or proportional integral (PI), and/or proportional integral 
derivative (PID) variety can be used for the correction step. More advanced control 
techniques may also be used to achieve tracking results. 

In addition to positioning a pointer on the display, tlie user may also 
select an item, usually represented by a menu entry or icon. A metliod for selecting or 
highlighting a specific item or icon on the display applies to both the real and the 
virtual display case. In some computer systems, simply positioning the mouse 
pointer on the item or the icon selects the item. Examples are with rollover items or 
web page links. In these cases, no additional method is required to highlight the item 
other than the positioning of the computer pointer upon it. 

In other cases, further user action is required to select or highlight an 
item. A common method of selecting or highlighting a specific item or icon using a 
conventional desktop mouse is by way of a single click of the mouse. With the 
system of this invention a method equivalent to this "single click" has to be defined. 
This method can be defined a priori or left for the user to define based on his/her 

convenience or taste. 

An example method for.a single click operation of the invention can 
be holding the pointing device steady over the item or in the vicinity of the item for a 
programmable length of time. For an example illustration of the method Figures 25a- 
25d show an example method for selecting or highlighting an item on tlie display. 
Because the pointer has been observed within the bounding box (dashed lines) of the 
icon for set number of frames (three frames in this case) the icon is selected. This 
amounts to a single click of the conventional computer mouse on the icon. To 
) accomplish this, the image or frames of images from the light sensor are observed for 
that length of time and if the pointing device (or the computer's pointer, or both) is 
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located over the item (or a tolerable distance from it) during that time, a command is 
sent to the computer to highlight or select the item. The parameters such as the 
applicable length of time and the tolerable distance can be further defined by the user 
during the set up or operation of tlie system as part of the options of the system. For 

5 the corresponding flowchart see Figure 26. The system first defines region around 

item or icon that will be used to determine if a single click is warranted (step 1 50). In 
step 152 the system defines the number of fi-ames or lengtli of time that the pointer 
has to be in the region to highlight the item. In step 1 54 the system finds the pointer 
device and using the mapping between the display space and the computer display 
10 144 positions the computer mouse accordingly on the display and stores the mouse 
position in a stack. The system then checks whether the stack is flill (step 1 56), If 
the stack is not fulL the system returns to step 154. If the stack is full, the system 
examines the stored mouse positions to determine whether they are all inside tlie 
bounding box around the item or the icon (step 158). The system checks if positions 

1 5 are all inside (step 1 60). If yes. the system then can highlight the item (step 1 64) and 
clear stack (step 166) before returning to step 154. If the positions are not all inside, 
the user can throw out the oldest mouse coordinate from the stack (step 162), and 

then return to step 1 54. 

Another example is to define a pointing device symboL stroke, or 
20 motion pattern, which can also be identified by the system by accumulating the 
positions at which the pointing device (or the computer pointer, or both) was 
observed. For example, drawing a circle around the item or underlining the item with 
the pointing device can be the "pointer symbol" that selects that item. To accomplish 
this, the image or frames of images fironi the light sensor are observed for an 
25 appropriate length of time and the paUi of the pointing device is analyzed to decide 
whether it forms a circle or if it underlines an icon or item on the display. A 
procedure similar to that outlined in Figure 26 can be used, this time to analyze the 
relationship of or tlie shape defined by the points at which the pointing device (or the 
computer pointer, or botii) has been observed. The speed with which such strokes 
30 must be carried out can also be defined by the user much the same way that a user can 
vary the double click speed of a conventional desktop mouse. 
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In most current computers, this positioning highlights the selected 
item, or changes its foreground and background color scheme of the said item to 
indicate that it has been selected. Note that this selection does not necessarily mean 
that the process or the program associated with that item has been activated. Such 
5 activation is discussed hereafter. 

The method for activating a specific process, program, or menu item 
represented on the display applies to both the real and the virtual display case. 

In addition to positioning a pointer on the display and selecting an 
item, one may also activate a process, a program or menu item represented on the 
1 0 display. In some computer systems or in certain programs or various locations of the 
desktop of a computer, simply a single click of the mouse button, as discussed 
regarding a method for selecting or highlighting a specific item or icon on the display 
on the item activates the program or the process defined by the item. Examples are 
web page links, many common drawing menu items, such as paintbrushes, and the 
1 5 shortcuts at the task bar of the Windows95 or Windows98 desktop. In these cases, no 
additional method is required to activate a process or a program otlier than that which 
is required for selecting or highlighting an item. 

A common method of activating a program or process using a 
conventional desktop computer mouse is by way of a double clicking of the mouse 
20 button. With the system of this invention a method equivalent to this "double click" 
has to be defined. This method can be defined a priori or during tlie operation of the 
system. 

An example method for a double click operation can be holding the 
pointing device steady over the item or in the vicinity of the item for a programmable 

25 length of time. This can be coordinated with the same type of method described in 
the previous section for a single mouse click. After holding the pointing device 
steady over an item for the lengUi of time required to define a single mouse click has 
elapsed and consequently a command for a single mouse click is in fact sent to the 
computer, holding the pointing device steady for additional length of time can send a 

30 second subsequent "click" to the computer, which when done with a certain time 
after the first such command, would constitute a "double click." This procedure is 



BNSDOCIO: <WO ^0152230A1J_> 



wo 01/52230 ^ ^ . PCT/USOl/00776 

24 

currently used by conventional computers, i.e., there is not necessarily a "double 
click" button on the conventional computer mouse. A double click is defined by two 
single clicks, which occur witliin a set number of seconds of each other. The length 
of time between two clicks can be set by the user using the conventional mouse 
5 program already installed on the computer. 

In addition to defining a "double click" as two closely spaced single 
clicks, one can define a pointing device symboL stroke or motion pattern to signify a 
double click. This pattern, too, can be identified by the system by accumulating the 
positions at which the pointing device was observed. For example, drawing a circle 
10 around the item could signily a double click whereas underlining the item with the 

pointing device could signify a single click. As before, to accomplish this, the image 
or frames of images from the light sensor are observed for an appropriate length of 
time and the path of the pointing device is analyzed to decide whether it forms a 
circle or if it underlines an icon or item on the display. The speed with which such 
1 5 strokes must be carried out can also be defined by the user much the same way that a 
user can vai7 tlie double click speed of a conventional desktop mouse. 

The common PC mouse has two to three buttons, which respond to 
single or double clicks in different ways. There are also ways of using the pointer as 
a drawing, a selecting/highlighting or a dragging tool, for example, by holding down 
20 the mouse button. The more recent PC mouse devices also have horizontal or 
vertical scroll wheels. Using tlie system of this invention, the many functions 
available from the common PC mouse (as well as other functions that may be made 
available in the future) can be accomplished with only an ordinary pointing device. 
To accomplish this, one can identify associate a series of other types of commands 
25 that one commonly carries out with a conventional mouse, such as scroll (usually 
accomplished with a wheel on a conventional mouse), move or polygon edit 
(commonly accomplished with the middle mouse button on a conventional mouse), 
display associated menus with an item (usually accomplished by clicking the right 
button on a conventional mouse), as well as a myriad of other commands, with a 
30 series of pointer device strokes, symbols, or motion patterns. This association may be 
built into the system a priori, or defined or refined by tlie user during the use of the 
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system. The association of strokes, symbols, or motion patterns using the pointing 
device is in a way analogous to the idea of handwritten character recognition used on 
a handheld computer with a stylus. The pressure sensitive pad on the handheld 
computer tracks and recognizes the strokes of the stylus. Similarly, the system of this 
invention can track and recognize the symbol that the pointing device is tracing in tlie 
real or virtual display space. 

The types and numbers of pointing device strokes can be traded 
against tlie richness of display menu items. For example, one can define a pointer 
stroke or symbol such for scrolling down a screen (e.g.. dragging the pointer device 
from top to bottom) or as simply another menu item, such as a forward button, on the 
display. In essence the pointer device can completely replicate all the functionality of 
the traditional PC mouse. It may also work with tlie traditional PC mouse in a 

complementary fashion. 

The system of this invention can also be interfaced with external 
systems, such as those that are voice or touch activated, otlier buttons on the pointing 
device that communicate to the computer to carry out a single or double click, or 
some other operation. In these cases, the system would still define over which item 
or display region the said operation will be carried out. but the operation itself is 
communicated by another system. Imagine for example, bringing the pointing device 
over a menu button and then tapping the display (where a touch or tap sound 
detecting system sends a "click" command to the computer) or saying "click" or 
"click click" (where a voice activated system sends the appropriate command to the 
computer). During the whole time, the system of this invention defines the computer 
display coordinates over which the command is carried out. 

; Hereinafter is a discussion for a metliod for writing, scribing, drawing, 

highlighting, annotating, or otherwise producing marks on the display. So far the 
description of the metliods of the invention have concentrated on selecting and 
activating processes associated with the menus or icons on the display. There are 
also many occasions on which the user would like to write or draw on the display in a 

[) more refined manner than one generally could with an ordinary mouse. There are 
many types of commercially available drawing tablets that one can attach to a 
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conventional computer for this purpose. The system of this invention can be used to 
accomplish the same. Furthermore, the system of this invention can also function as 
an electronic whiteboard that can transmit to a computer the marks upon it. In 
contrast to the case with electronic white boards, when the system of this invention is 
used, no expensive special writing board is required. 

Figure 27 shows a light pen 170 that can be used successfully with the 
system of this invention both as a pointing device and a drawing and writing 
instrument. The light pen 170 could be activated by holding down the power button 
172 or by applying some pressure to its tip 174. Thus when the light pen 170 is 
pressed against a board on which the computer display is projected, the tip 1 74 would 
light up and would become easily identifiable to the system. Its light can be traced to 
form lines or simply change the color of the pixels it touches upon. The system can 
be interfaced witli common drawing programs which allow the user to define a set of 
brush colors, lines, drawing shapes and other functions (e.g., erasers, smudge tools, 
etc.) that enrich the works of art the user can thus create. 

Note that throughout no actual mark is made on the display or the 
projection space. Moreover, no actual multi-colored pens or unique screen or surface 
are required. The same system could also be. used on a blank board to capture 
everything the user wiites or draws with the light pen. Because the light pen stylus is 
designed to function like a witing device, the user may convenienth' and easily 
scribe notes directly onto the display without having to access a PC keyboard or 
target a sensor in order to annotate a presentation. 

The annotations become part of tlie projected document as the user 
creates them since the presentation or drawing program adds them to the document 
that the user is creating almost instantaneously. The computer interfaced with the 
display in turn puts the resulting document to the display space. Furthermore, with 
the same LED stylus, the user may navigate through any other program or document. 
Best of all, this stylus capability can be a built-in feature of the overall system 
including the pointing functions. No additional special software is required since the 
) system simply functions as a mouse or stylus at the same time. Other types of 
pointing devices can also be used for the same purpose. 
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Imagine as a potential application an instructor before a projected 
display. He or she is using the light pen to draw on the ordinary wall or surface onto 
which the display is projected using an LCD projector. Imagine that the training 
session contains an electronic training document as well as notes and illustrations 

5 scribbled by the instructor during the training session. Imagine again that all those 
notes and illustrations the instructor makes can be recorded as the instructor makes 
them on the board with the Ught pen. The final annotated document can be 
electronically stored and transmitted anywhere. The result is a superb instant 
videoconferencing, distance learning, documentation and interaction tool. 

1 0 The same system can also be used for text entr>' - if tlie strokes can be 

recognized as letters or characters. This again is similar to the case where tlie strokes 
of the stylus on the pressure-sensitive writing area can be recognized as letters or 
characters. 

The described method for writing, scribing, drawing, highlighting, 
1 5 annotating, or otherwise producing marks on the display mostly applies to tlie real 
display case. Despite that, some simple shapes can be drawn on a virtual display 
space. Since the user will immediately view the rendition or results of his/lier marks, 
he/she can adjust the strokes of the pointing device accordingly. 

Finally, in Figure 28, a system operation flowchart is included to 
20 summarize the background or backbone process of the system. The system proceeds 
to system operation 180 from step 142. In step 182, the system acquires data from 
the sensor or one or more image frames from tlie light sensor or camera. In step 184, 
the system locates the pointing device. This step usually requires analysis of the data 
or image frame or frames acquired in 182. The analysis is made by using the 
25 distinguishing characteristics of the pointing device against the real display 88 or the 
same against the virtual display 120. If the system fails to locate the pointing device, 
it will go back to step 182. If it locates the pointing device it will move to step 186. 
In step 1 86, the system maps the position of pointing device to a point on the real or 
virtual display space. Especially if the pointing device spans over multiple regions, 
30 this step may require that the borders of the pointing device, or its center of gravity be 
identified. In step 1 88, the system finds the computer display position corresponding 
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to the pointing device position. Tliis step requires tlie mapping between the display 
space and the computer display 144. In step 190, the system positions the computer's 
pointing icon (e.g., mouse arrow) at the computed computer display position. Note 
that the step 190 may be skipped or suppressed if the system is engaged in another 

5 task or has been programmed not to manipulate the computer's pointing icon. 

Methods for implementing the functions normally associated with a 
computer mouse, e.g., selecting an item on the display, starting a process associated 
with an item on the display, dragging or moving objects across the display, drawing 
on the display, scrolUng across the display, are processes that emanate from this 

10 backbone process, in particular from steps 186, 188. and/or 190. 
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What is claimed is: 

1 1 . A system for interacting with displays and all devices that use 

2 such displays comprised of 

3 a. a display, 

4 b. a sensor or camera, 

5 c. a pointing device that can be registered by the sensor or 

6 camera, 

7 d. a method for detecting the pointing device, 

8 e. a method for establishing the mapping between the position of 

9 the pointing device and a corresponding location on the display. 

1 2. A system according to claim 1 wherein tlie. sensor or camera. 

2 in addition to registering the image of the pointing object, can also register at least 

3 one of (i) the image of the display and (ii) the reflection or effect that the pointing 

4 device can produce on the display. 

1 3. A system as defined by claim 1 which commands the 

2 positioning of a pointing icon on the display. 

1 4. A system according to claim 1 wherein the pointing device is a 

2 part of the human body such as a hand or a finger, or an ornament or device worn on 

3 the human body such as a glove or thimble. 

J 5. A system according to claim 1 wherein the pointing device is 

2 used to point to regions of the display by way of changing its position, 

3 attitude, or presentation. 

1 6. A system according to claim 1 wherein the pointing device is 

2 used to define a particular point or region on the display. 
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1 7. A system according to claim 1 wherein the pointing device is 

2 used to define a vector on the plane of the display that indicates a direction 

3 and magnitude relative to or with respect to an item on tlie display or a 

4 region of the display. 

1 8. A system according to claim 3 wherein the pointing icon on 

2 the display can be registered by the sensor or camera. 

1 9. A system according to claim 8 which also includes a method 

2 for correcting the offsets between (i) tlie position of the pointing device, or reflection, 

3 or effect thereof on the display as observed by the user or by the sensor or the camera, 

4 and (ii) the position of the pointer icon on the display. 

1 10. A system as defined by claim 1 which also includes at least 

2 one of the following: 

3 a. a method for selecting or highlighting a specific item or icon 

4 on the display, 

5 b. a method for activating a specific process, program, or menu 

6 item represented on the display, and 

7 c. a method for writing, scribing, drawing, highlighting, 

8 annotating, or otherwise producing marks on the display. 

1 1 1 . A method for detecting the pointing device comprising 

2 a. retrieval of data or image from a sensor or camera, and 

3 b. analysis of the data or image from the sensor or camera to 

4 locate the pointing device in the data, or locating at least a set of the picture elements 

5 in the image that comprise the rendition of the pointing device. 

1 12. A method according to claim 1 1 wherein the characteristics 

2 that distinguish the pointing device from other objects in the data from the 
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sensor or the image from the camera are known a priori. 



1 1 3 . A method according to claim 1 1 wherein the characteristics 

2 that distinguish the pomting device from other objects in the data from the sensor or 

3 the image from the camera are determined based analysis of at least one set of the 

4 data acquired from the sensor or one image acquired from the camera. 



1 14. A method according to claim 13 wherein the characteristics 

2 that distinguish the pointing device from other objects, whose rendition are present in 
the data from the sensor or in the image from the camera, is obtained by 

4 a. acquiring at least two sets of data from the sensor or at least 

5 two images from the camera, one with the pointing device in view of the sensor 

6 or the camera and one without, and 

7 b. comparing the two sets with one another. 



1 1 5. A method according to claim 1 1 wherein adjustments or 

2 modifications are made to the position, sensitivity, and other settings of the sensor or 

3 the camera pursuant the analysis of the data or image retrieved from the sensor or tlie 

4 camera. 

1 1 6. A method according to claim 1 1 wherein at least part of the 

2. procedures for the method is carried out using at least in part the computing 

3 mechanisms available on one or more of the following: the display, or the sensor or 

4 camera, or the pointing device, or the device producing the signal shown on the 

5 display, or the device producing the pointing icon on the display. 

1 1 7. A method for establishing the mapping between the set of 

2 positions that a pointing device can take and the set of corresponding locations on the 

3 display comprising; 

4 a. defining the range of positions that the pointing device can 



BNSDCXJID: <WO ^0152230A1_L> 



wo 01/52230 ^ A PCT/USOl/00776 



32 



5 assume, 

6 b. defining the boundaries of the range of positions that the 

7 pointing device can take with geometric representations, 

8 c. transforming the geometric representation of the arrangement 

9 of regions on the display so that it fits optimally into the boundaries of tlie range of 
10 positions that the pointing device can take. 

1 1 8. A method according to claim 1 7 wherein the range of positions 

2 that the pointing device may assume is defined by querying the user to point to a set 

3 of points on the display. 

1 1 9. A method according to claim 1 8 wherein the range of positions 

2 that the pointing device can assume is defined by the boundaiy contours of the 

3 display as they are registered by the sensor or tlie camera. 

1 20. A method according to claim 19 wherein at least one special 

2 display image is used to establish the mapping between the positions that a pointing 

3 device can take and a corresponding locations on the display. 

1 2 1 . A method according to claim 1 7 wherein at least part of tlie 

2 procedures for the method is carried out using at least in part the computing 

3 mechanisms available on one or more of the following: the display, or the sensor or 

4 camera, or the pointing device, or the device producing the signal shown on the 

5 display, or the device producing tlie pointing icon on the display. 

1 22. A method for detecting the display comprising 

2 a. retrieval of data or image from a sensor or camera, and 

3 b. analysis of the data or image from the sensor or camera to 

4 locate the display in the data, or locating at least a set of the picture elements in 

5 the image that comprise the rendition of the display in the image. 
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1 23. A method according to claim 22 wherein the characteristics 

2 that distinguish the display from other objects in the data from the sensor or the 

3 image from the camera are known a priori. 

1 24. A method according to claim 22 wherein the characteristics 

2 that distinguish the display from other objects in the data from the sensor or the 

3 image from the camera are detennined based on analysis of at least one set of the data 

4 acquired from the sensor or one image acquired from the camera. 

1 25. A method according to claim 22 wherein the display refers to 

2 the range of positions that the pointing device can take. 

1 26. A method according to claim 24 wherein the characteristics 

2 that distinguish the display from other objects, whose rendition are present in the data 

3 from the sensor or in the image from the camera, is obtained by 

4 a. acquiring at least two sets of data from the sensor or at least 

5 two images from the camera, one with the display off in view of the sensor or the 

6 camera and one with the display on, and 

7 b. comparing the two sets with one another. 

1 27. A method according to claim 22 wherein adjustments or 

2 modifications are made to the position, sensitivity, and other settings of the sensor or 

3 the camera pursuant the analysis of the data or image retrieved from the sensor or the 

4 camera. 

1 28. A method according to claim 22 wherein at least part of the 

2 procedures for the method is carried out using at least in part the computing 

3 mechanisms available on one or more of the following: the display, or the sensor or 

4 camera, or the pointing device, or the device producing the signal shown on the 
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display, or the device producing the pointing icon on the display. 
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AMENDED CLAIMS 
[received by the International Bureau on 16 May 2001 (16.05.01); 
original claims 3- 7, 10, 15, 16, 21,25,27 and 28 amended; remaining claims unchanged (5 page)] 



1 1 . A system for interacting with displays and all devices that 

2 use such displays comprised of 

3 a. a display, 

4 b. a sensor or camera 

5 c. a pointing device that can be registered by the sensor or 

6 camera, 

7 d. a ,method for detecting the pointing device, 

8 e. a method for establishing the mapping between the position 



of the pointing device and a corresponding location on the display. 



1 2. A system according to claim 1 wherein the sensor or 

2 camera, in addition to registering the image of the pointing object, can also 

3 register at least one of (i) the Image of the display and (ii) the refelction or effect 

4 that the pointing device can produce on the display. 

1 3. A system as defined by one of the claims 1-2 which 

2 commands the positioning of a pointing icon on the display. 

1 4. A system according to one of the claims 1-3 wherein the 

2 pointing device is a part of the human body such as a hand or a finger, or an 

3 ornament or device worn on the human body such as a glove or thimble. 

1 5, A system according to one of the claims 1-4 wherein the 

2 pointing device is used to point to regions of the display by way of changing its 

3 position, attitude, or presentation. 

1 6. A system according to one of the claims 1-5 wherein the 

2 pointing device is used to define a particular point or region on the display. 
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1 7. A system according to one of the claims 1-5 wherein the 

2 pointing device is used to define a vector on the plane of the display that 

3 indicates a direction and magnitude relative to or with respect to an item on the 

4 display or a region of the display. 

1 B. A system according to claim 3 wherein the pointing icon on 

2 the display can be registered by the sensor or camera. 

1 9. A system according to claim 8 which also includes a 

2 method for correcting the offsets between (i) the position of the pointing device, 

3 or reflection, or effect thereof on the display as observed by the user or by the 

4 sensor or the camera, and (ii) the position of the pointer icon on the display. 

1 10. A system as defined by one of the claims 1-9 which also 

2 includes at least one of the following: 

3 a. a method for selecting or highlighting a specific item or icon 

4 on the display, 

5 b, a method for activating a specific process, program, or 

6 menu item represented on the display, and 

7 c. a method for writing, scribing, drawing, highlighting, 

8 annotating, or otherwise producing marks on the display. 

1 1 1 . A method for detecting the pointing device comprising 

2 a. retrieval of data or image from a sensor or camera, and 

3 b. analysis of the data or image from the sensor or camera to 

4 locate the pointing device in the data, or locating at least a set of the picture 

5 elements in the image that comprise the rendition of the pointing device. 

1 12. A method according to claim 1 1 wherein the characteristics 

2 that distinguish the pointing device from other objects in the data from the 
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sensor or the image from the camera are known a priori. 

13. A method according to claim 1 1 wherein the characteristics 
that distinguish the pointing device from other objects in the data from the 
sensor or the image from the camera are determined based analysis of at least 
one set of the data acquired from the sensor or one image acquired from the 
camera. 

14. A method according to claim 13 wherein the characteristics 
that distinguish the pointing device from other objects, whose rendition are 
present in the data from the sensor or in the image from the camera, is obtained 
by 

a. acquiring at least two sets of data from the sensor or at 
least two images from the camera, one with the pointing device In view of the 
sensor or the camera and one without and, 

b. comparing the two sets with one another. 

15. A method according to one of the claims 1 1-14 wherein 
adjustments or modifications are made to the position, sensitivity, and other 
settings of the sensor or the camera pursuant the analysis of the data or Image 
retrieved from the sensor or the camera. 

16. A method according to one of the claims 11-15 wherein at 
least part of the procedures for the method is carried out using at least in part 
the computing mechanisms available on one or more of the following: the 
display, or the sensor or camera, or the pointing device, or the device producing 
the signal shown on the display, or the device producing the pointing icon on 
the display. 

1 7. A method for establishing the mapping between the set of 
positions that a pointing device can take and the set of corresponding locations 
on the display comprising: 

a. defining the range of positions that the pointing device can 
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J> assume. 

6 b. defining the bo jndaries of the range of positions that the 

pointing device can take with geometric representations, 
S c. transforming the geometric representation of the 

9 arrangement of regions on the display so that it fits optimaJly into the boundaries 

^ the range of positions that the pointing device can take. 

1 18. A method according to claim 17 wherein the range of 

2 positions that the pointing device may assume Is defined by querying the user 

3 to point to a set of points on the display. 

1 19. A method according to claim 18 wherein the range of 

2 positions that the pointing device can assume is defined by the boundary 

3 contours of the display as they are registered by the sensor or the camera. 

1 20. A method according to claim 1 9 wherein at least one 

2 special display image is used to establish the mapping between the positions 

3 that a pointing device can take and a corresponding locations on the display. 

1 2 1 . A method according to one of the claims 1 7-20 wherein at 

2 least part of the procedures for the method is carried out using at least in part 

3 the computing mechanisms available on one or more of the following: the 

4 display, or the sensor or camera, or the pointing device, or the device producing 

5 the signal shown on the display, or the device producing the pointing icon on 

6 the display. 

1 22. A method for detecting the display comprising 

2 a. retrieval of data or image from a sensor or camera, and 

3 b. analysis of the data or image from the sensor or camera to 

4 locate the display in the data, or locating at least a set of the picture elements in 

5 the image that comprise the rendition of the display in the image. 
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T 23. A method according to claim 22 wherein the characteristics 

2 that distinguish the display from other objects in the data from the sensor or the 

3 image from the camera are known a priori. 

1 24. A method according to claim 22 wherein the characteristics 

2 that distinguish the display from other objects in the data from the sensor or the 

3 image from the camera are determined based on analysis of at least one set of 

4 the data acquired from the sensor or one image acquired from the camera. 

1 25 A method according to one of the claims 22 to 26 wherein 

2 the display refers to the range of positions that the pointing device can take. 

1 26. A method according to claim 24 wherein the characteristics 

2 that distinguish the display from other objects, whose rendition are present in 

3 the data from the sensor or In the image from the camera, is obtained by 

4 a. acquiring at least two sets of data from the sensor or at 

5 least two images from the camera, one with the display off in view of the sensor 

6 or the camera and one with the display on, and 

7 comparing the two sets with one another. 

1 27. A method according to one of the claims 22-26 wherein 

2 adjustments or modifications are made to the position, sensitivity, and other 

3 settings of the sensor or the camera pursuant the analysis of the data or Image 

4 retrieved from the sensor or the camera. 

1 28. A method according to one of the claims 22-27 wherein at 

2 least part of the procedures for the method is carried out using at least in part 

3 the computing mechanisms available on one or more of the following: the 

4 display, or the sensor or camera, or the pointing device, or the device producing 

5 the signal shown on the 
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STATEMENT UMPER ARTICLE 19(1) 

The amendments to claims 3, 4, 5, 6, 7, 10. 15, 16, 21, 25, 27 and 
28 submitted herewith are to make the aforementioned claims multiple 
dependent claims. 
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FIG - 10 
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FIG - 14 



START 



1/ 



50 



52 



£ 



Turn display on, using the image 
from the light sensor, find display 
area based on the characteristics 
of the display or the known image 
on the display 



Turn display off. with the sensor, 
capture a frame of the display 



6'' 



56- 



i 



Turn display on. capture a frame 
of the display space 



Locate display by examining the 
difference between the two frames 



60 



Adjust the light sensor position and sensing 
parameters for the best viewing conditions 



58 




Define the borders of the display in the image captured 
by the light sensor as continuous lines or curves 



Borders of the display as it is 
^captured by the light sensor, their 
visual characteristics, location and 
curvature 



Continue to pointing device 
registration (Figure 15) 



66 



68 



SUBSTITUTE SHEET (RULE 26) 



wo 01/5223i 



PCTAJSOl/00776 



FIG - 15 



9/19 



70 



Instruct user to register the pointing device he/she will use. The user may select a 
pointing device from a list or allow the system to view the pointing device by 
allowing the pointing device to be viewed by the light sensor. 
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Instruct user to point the 
pointing device to various 
points in the display. Capture 
one or more frames of the 
display space with the 
pointing device 



Capture a frame of the display space 
without the pointing device 



Capture a frame of the display space 
with the pointing device 



Locate pointing device by examining 
the difference between the two 




78 



Adjust the light sensor position and sensing 
parameters for the best viewing conditions 




Determine the distinguishing characteristics of the pointing device 
as opposed to the rest of the display by obsen/ing it via the light 

sensor against any image on the display or against a set of 
calibration images. Adjust the light sensor position and sensing 
parameters for optimum operation. 



86 



Distinguishing characteristics 
of the pointing device against 
a variety of display backgrounds. 

88 



Continue to computing the mapping 
between the display space registered 
by the light sensor and the computer 
display 
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FIG - 16 



START 



90 



92 



Turn display on, instruct the 
user to point the pointing device 
to a convenient location of the 
display (e.g., center). Using the 
image from the light sensor, find 
the pointing device based on the 
known characteristics of the 
pointing device. 



JL 
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Instruct user to hide pointing device, with 
the light sensor, capture a frame of the 
display space 



96 



Instruct user to point the pointing device to a 
convenient location of the display, capture a 

frame of the display space 
1 ' 



Locate the pointing device by examining 
the difference between the two frames 
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98 



Adjust the light sensor position and sensing 
parameters for best viewing conditions 



104 




Instruct user to point with the pointing device to the borders 
and/or various locations of the display. Capture frames 
with light sensor. . 
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Define the borders of the display space in the image captured 
by the light sensor as continuous lines or curves 
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108 



Borders of the display as it is 
^captured by the light sensor, their 
visual characteristics, location and, 
curvature 



Continue to pointing device 
registration (Figure 17) 
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FIG - 17 



112 



Instruct user to point with the pointing device to the borders and/or various 
locations of the display. Capture frames with light sensor. 



114 



Adjust the light sensor position and sensing 
parameters for the best viewing conditions 




Determine the distinguishing characteristics of the pointing device 
as opposed to the rest of the display by obsenying it via the light 
sensor against the background of the virtual display. Adjust the 
light sensor'position and sensing parameters for optimum ooeration. 
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Continue to computing the mapping 
between the display space registered 
by the light sensor and the computer 

disDiay 



120 



Distinguishing charactenstics 
of thel^oinring device against 
a variety of display backgrounds. 
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FIG - 18 



130 




Z Borders of the display as it is 
captured by the light sensor, 
their visual characteristics, 
location and curvature / 



132 




Divide the display space into the 
same number of regions as the 
those of the computer display 
using the information from the 
information from the borders of 



the display space. 



134" 
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Distinguishing characteristics 
of the pointing device (against^ 
a variety of display 
backgrounds if real). 



Images capture with the 
pointing device pointing to 
various regions of the display. 



140 



Establish correspondence between 
the disolay space observed by the 
light sensor and the computer display 
region by region. Make necessan/ 
adjustments to the boundanes of 
individual regions as necessary. 



138 



Using the information from the 
position of the pointing device 
previously registered by the user 
and the images captured when 
the user pointed the pointing 
device to the regions of the 
display as instructured by the 
mapping betvyeen the image 
space registered by the light 
sensor and the computer display. 



142 



Continue to system operation 



144 




Mapping between the display 
space and the computer 
display 
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FIG - 24A 
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FIG - 26 



150 



Define region around item or icon 
that will be used to determine if a 
single click is warranted 



152 



Define number of frames or length 
of time that the pointer has to be in 
the region to highlight the item 



Find pointer device and position 
the computer mouse accordingly 
on the display, store mouse 
position is stack 



154 




1 { 



164 



highlight item. 



Clear stack 



166 



Examine stored mouse positions to 
determine whether they are all inside 
the bounding box around the item 
or the icon. 



160 




Yes 



No 



162 



Throw out the oldest mouse 
coordinate from the stack. 



SUBSTITUTE SHEET (RULE 26) 



wo 01/52230 



PCT/USOl/00776 



19/19 



-180 



FIG - 28 



System 
Operation 



Get data from sensor or 
image frame from camera 



Locate pointing device 



185 



Yes 



Map position of pointing 
device to point on the real 
or virtual display space 



Find the computer display 
position corresponding to 
the pointing device position 



Position the computer's 
pointing icon (e.g., mouse 

arrow) at the computed 
computer display position 
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